Fast Offline Policy Optimization for Large Scale Recommendation

Personalised interactive systems such as recommender require selecting relevant items from massive catalogs dependent on context. Reward-driven offline optimisation of these can be achieved by a relaxation the discrete problem resulting in policy learning or REINFORCE style algorithms. Unfortunately, this step requires computing sum over entire catalogue making complexity evaluation gradient (and hence each stochastic descent iterations) linear size. This calculation is untenable many real world examples large systems, severely limiting usefulness method practice. In paper, we derive an approximation algorithms that scale logarithmically with Our contribution based upon combining three novel ideas: new Monte Carlo estimate policy, self normalised importance sampling estimator and use fast maximum inner product search at training time. Extensive experiments show our algorithm order magnitude faster than naive approaches yet produces equally good policies.